{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Traditional_transfer_learning.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3",
      "language": "python"
    },
    "language_info": {
      "name": "python",
      "version": "3.7.6-final"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "icVPttAPFurS"
      },
      "source": [
        "# Traditional transfer learning tutorial\n",
        "This is a tutorial notebook for traditional transfer learning (i.e., non-deep learning).\n",
        "\n",
        "We'll implement two algorithms:\n",
        "- **TCA** (Transfer Component Analysis) [1]\n",
        "- **BDA** (Balanced Distribution Adaptation) [2]\n",
        "\n",
        "Then, we test the algorithms using **Office-Caltech10** SURF dataset. This dataset will be downloaded automatically in this tutorial.\n",
        "\n",
        "References:\n",
        "\n",
        "[1] Pan S J, Tsang I W, Kwok J T, et al. Domain adaptation via transfer component analysis[J]. IEEE Transactions on Neural Networks, 2010, 22(2): 199-210.\n",
        "\n",
        "[2] Wang J, Chen Y, Hao S, et al. Balanced distribution adaptation for transfer learning[C]//2017 IEEE international conference on data mining (ICDM). IEEE, 2017: 1129-1134."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fs_zONUnFurW"
      },
      "source": [
        "## Download and unzip dataset\n",
        "You can also download the dataset from here: https://github.com/jindongwang/transferlearning/tree/master/data#office-caltech10"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UItzY-rN-C6Z",
        "outputId": "fe3bb35e-292f-48bb-db11-385fc1e53191"
      },
      "source": [
        "!wget https://transferlearningdrive.blob.core.windows.net/teamdrive/dataset/office-caltech-surf.zip\n",
        "!unzip office-caltech-surf.zip"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-03-30 12:27:06--  https://transferlearningdrive.blob.core.windows.net/teamdrive/dataset/office-caltech-surf.zip\n",
            "Resolving transferlearningdrive.blob.core.windows.net (transferlearningdrive.blob.core.windows.net)... 20.150.17.228\n",
            "Connecting to transferlearningdrive.blob.core.windows.net (transferlearningdrive.blob.core.windows.net)|20.150.17.228|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2487903 (2.4M) [application/x-zip-compressed]\n",
            "Saving to: ‘office-caltech-surf.zip.1’\n",
            "\n",
            "office-caltech-surf 100%[===================>]   2.37M  1.13MB/s    in 2.1s    \n",
            "\n",
            "2021-03-30 12:27:09 (1.13 MB/s) - ‘office-caltech-surf.zip.1’ saved [2487903/2487903]\n",
            "\n",
            "Archive:  office-caltech-surf.zip\n",
            "  inflating: amazon_surf_10.mat      \n",
            "  inflating: caltech_surf_10.mat     \n",
            "  inflating: dslr_surf_10.mat        \n",
            "  inflating: webcam_surf_10.mat      \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4wSXVFUZFurX"
      },
      "source": [
        "## Import libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fRoy6hvY-zKM"
      },
      "source": [
        "import numpy as np\n",
        "import scipy.io\n",
        "import scipy.linalg\n",
        "import sklearn.metrics\n",
        "from sklearn.neighbors import KNeighborsClassifier"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3fuq1sqFurY"
      },
      "source": [
        "## Define a kernel function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FAmo6hc_-5Pu"
      },
      "source": [
        "def kernel(ker, X1, X2, gamma):\n",
        "    K = None\n",
        "    if not ker or ker == 'primal':\n",
        "        K = X1\n",
        "    elif ker == 'linear':\n",
        "        if X2 is not None:\n",
        "            K = sklearn.metrics.pairwise.linear_kernel(np.asarray(X1).T, np.asarray(X2).T)\n",
        "        else:\n",
        "            K = sklearn.metrics.pairwise.linear_kernel(np.asarray(X1).T)\n",
        "    elif ker == 'rbf':\n",
        "        if X2 is not None:\n",
        "            K = sklearn.metrics.pairwise.rbf_kernel(np.asarray(X1).T, np.asarray(X2).T, gamma)\n",
        "        else:\n",
        "            K = sklearn.metrics.pairwise.rbf_kernel(np.asarray(X1).T, None, gamma)\n",
        "    return K"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GhngbY7DFurY"
      },
      "source": [
        "# Implement TCA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yjaBHq7P-8qB"
      },
      "source": [
        "class TCA:\n",
        "    def __init__(self, kernel_type='primal', dim=30, lamb=1, gamma=1):\n",
        "        '''\n",
        "        Init func\n",
        "        :param kernel_type: kernel, values: 'primal' | 'linear' | 'rbf'\n",
        "        :param dim: dimension after transfer\n",
        "        :param lamb: lambda value in equation\n",
        "        :param gamma: kernel bandwidth for rbf kernel\n",
        "        '''\n",
        "        self.kernel_type = kernel_type\n",
        "        self.dim = dim\n",
        "        self.lamb = lamb\n",
        "        self.gamma = gamma\n",
        "\n",
        "    def fit(self, Xs, Xt):\n",
        "        '''\n",
        "        Transform Xs and Xt\n",
        "        :param Xs: ns * n_feature, source feature\n",
        "        :param Xt: nt * n_feature, target feature\n",
        "        :return: Xs_new and Xt_new after TCA\n",
        "        '''\n",
        "        X = np.hstack((Xs.T, Xt.T))\n",
        "        X /= np.linalg.norm(X, axis=0)\n",
        "        m, n = X.shape\n",
        "        ns, nt = len(Xs), len(Xt)\n",
        "        e = np.vstack((1 / ns * np.ones((ns, 1)), -1 / nt * np.ones((nt, 1))))\n",
        "        M = e * e.T\n",
        "        M = M / np.linalg.norm(M, 'fro')\n",
        "        H = np.eye(n) - 1 / n * np.ones((n, n))\n",
        "        K = kernel(self.kernel_type, X, None, gamma=self.gamma)\n",
        "        n_eye = m if self.kernel_type == 'primal' else n\n",
        "        a, b = np.linalg.multi_dot([K, M, K.T]) + self.lamb * np.eye(n_eye), np.linalg.multi_dot([K, H, K.T])\n",
        "        w, V = scipy.linalg.eig(a, b)\n",
        "        ind = np.argsort(w)\n",
        "        A = V[:, ind[:self.dim]]\n",
        "        Z = np.dot(A.T, K)\n",
        "        Z /= np.linalg.norm(Z, axis=0)\n",
        "        Xs_new, Xt_new = Z[:, :ns].T, Z[:, ns:].T\n",
        "        return Xs_new, Xt_new\n",
        "\n",
        "    def fit_predict(self, Xs, Ys, Xt, Yt):\n",
        "        '''\n",
        "        Transform Xs and Xt, then make predictions on target using 1NN\n",
        "        :param Xs: ns * n_feature, source feature\n",
        "        :param Ys: ns * 1, source label\n",
        "        :param Xt: nt * n_feature, target feature\n",
        "        :param Yt: nt * 1, target label\n",
        "        :return: Accuracy and predicted_labels on the target domain\n",
        "        '''\n",
        "        Xs_new, Xt_new = self.fit(Xs, Xt)\n",
        "        clf = KNeighborsClassifier(n_neighbors=1)\n",
        "        clf.fit(Xs_new, Ys.ravel())\n",
        "        y_pred = clf.predict(Xt_new)\n",
        "        acc = sklearn.metrics.accuracy_score(Yt, y_pred)\n",
        "        return acc, y_pred"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yuKeKRCUFurZ"
      },
      "source": [
        "## Implement BDA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wVb8DLYA_U03"
      },
      "source": [
        "class BDA:\n",
        "    def __init__(self, kernel_type='primal', dim=30, lamb=1, mu=0.5, gamma=1, T=10, mode='BDA', estimate_mu=False):\n",
        "        '''\n",
        "        Init func\n",
        "        :param kernel_type: kernel, values: 'primal' | 'linear' | 'rbf'\n",
        "        :param dim: dimension after transfer\n",
        "        :param lamb: lambda value in equation\n",
        "        :param mu: mu. Default is -1, if not specificied, it calculates using A-distance\n",
        "        :param gamma: kernel bandwidth for rbf kernel\n",
        "        :param T: iteration number\n",
        "        :param mode: 'BDA' | 'WBDA'\n",
        "        :param estimate_mu: True | False, if you want to automatically estimate mu instead of manally set it\n",
        "        '''\n",
        "        self.kernel_type = kernel_type\n",
        "        self.dim = dim\n",
        "        self.lamb = lamb\n",
        "        self.mu = mu\n",
        "        self.gamma = gamma\n",
        "        self.T = T\n",
        "        self.mode = mode\n",
        "        self.estimate_mu = estimate_mu\n",
        "\n",
        "    def fit_predict(self, Xs, Ys, Xt, Yt):\n",
        "        '''\n",
        "        Transform and Predict using 1NN as JDA paper did\n",
        "        :param Xs: ns * n_feature, source feature\n",
        "        :param Ys: ns * 1, source label\n",
        "        :param Xt: nt * n_feature, target feature\n",
        "        :param Yt: nt * 1, target label\n",
        "        :return: acc, y_pred, list_acc\n",
        "        '''\n",
        "        list_acc = []\n",
        "        X = np.hstack((Xs.T, Xt.T))\n",
        "        X /= np.linalg.norm(X, axis=0)\n",
        "        m, n = X.shape\n",
        "        ns, nt = len(Xs), len(Xt)\n",
        "        e = np.vstack((1 / ns * np.ones((ns, 1)), -1 / nt * np.ones((nt, 1))))\n",
        "        C = len(np.unique(Ys))\n",
        "        H = np.eye(n) - 1 / n * np.ones((n, n))\n",
        "        mu = self.mu\n",
        "        M = 0\n",
        "        Y_tar_pseudo = None\n",
        "        Xs_new = None\n",
        "        for t in range(self.T):\n",
        "            N = 0\n",
        "            M0 = e * e.T * C\n",
        "            if Y_tar_pseudo is not None and len(Y_tar_pseudo) == nt:\n",
        "                for c in range(1, C + 1):\n",
        "                    e = np.zeros((n, 1))\n",
        "                    Ns = len(Ys[np.where(Ys == c)])\n",
        "                    Nt = len(Y_tar_pseudo[np.where(Y_tar_pseudo == c)])\n",
        "\n",
        "                    if self.mode == 'WBDA':\n",
        "                        Ps = Ns / len(Ys)\n",
        "                        Pt = Nt / len(Y_tar_pseudo)\n",
        "                        alpha = Pt / Ps\n",
        "                        mu = 1\n",
        "                    else:\n",
        "                        alpha = 1\n",
        "\n",
        "                    tt = Ys == c\n",
        "                    e[np.where(tt == True)] = 1 / Ns\n",
        "                    yy = Y_tar_pseudo == c\n",
        "                    ind = np.where(yy == True)\n",
        "                    inds = [item + ns for item in ind]\n",
        "                    e[tuple(inds)] = -alpha / Nt\n",
        "                    e[np.isinf(e)] = 0\n",
        "                    N = N + np.dot(e, e.T)\n",
        "\n",
        "            # In BDA, mu can be set or automatically estimated using A-distance\n",
        "            # In WBDA, we find that setting mu=1 is enough\n",
        "            if self.estimate_mu and self.mode == 'BDA':\n",
        "                if Xs_new is not None:\n",
        "                    mu = estimate_mu(Xs_new, Ys, Xt_new, Y_tar_pseudo)\n",
        "                else:\n",
        "                    mu = 0\n",
        "            M = (1 - mu) * M0 + mu * N\n",
        "            M /= np.linalg.norm(M, 'fro')\n",
        "            K = kernel(self.kernel_type, X, None, gamma=self.gamma)\n",
        "            n_eye = m if self.kernel_type == 'primal' else n\n",
        "            a, b = np.linalg.multi_dot(\n",
        "                [K, M, K.T]) + self.lamb * np.eye(n_eye), np.linalg.multi_dot([K, H, K.T])\n",
        "            w, V = scipy.linalg.eig(a, b)\n",
        "            ind = np.argsort(w)\n",
        "            A = V[:, ind[:self.dim]]\n",
        "            Z = np.dot(A.T, K)\n",
        "            Z /= np.linalg.norm(Z, axis=0)\n",
        "            Xs_new, Xt_new = Z[:, :ns].T, Z[:, ns:].T\n",
        "\n",
        "            clf = sklearn.neighbors.KNeighborsClassifier(n_neighbors=1)\n",
        "            clf.fit(Xs_new, Ys.ravel())\n",
        "            Y_tar_pseudo = clf.predict(Xt_new)\n",
        "            acc = sklearn.metrics.accuracy_score(Yt, Y_tar_pseudo)\n",
        "            list_acc.append(acc)\n",
        "            print('{} iteration [{}/{}]: Acc: {:.4f}'.format(self.mode, t + 1, self.T, acc))\n",
        "        return acc, Y_tar_pseudo, list_acc\n"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5IuuTWGlFurc"
      },
      "source": [
        "## Load data\n",
        "We'll load data. For demonstration, we use *Caltech* as the source and *amazon* as the target domain."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aRBJ87xg-_H7"
      },
      "source": [
        "src, tar = 'caltech_surf_10.mat', 'amazon_surf_10.mat'\n",
        "src_domain, tar_domain = scipy.io.loadmat(src), scipy.io.loadmat(tar)\n",
        "Xs, Ys, Xt, Yt = src_domain['feas'], src_domain['label'], tar_domain['feas'], tar_domain['label']"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p1qkzPZxFure"
      },
      "source": [
        "## Test TCA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7U6Ez1B2Furf",
        "outputId": "47398327-4c65-41e9-fee0-1783fe562325"
      },
      "source": [
        "tca = TCA(kernel_type='linear', dim=30, lamb=1, gamma=1)\n",
        "acc, ypre = tca.fit_predict(Xs, Ys, Xt, Yt)\n",
        "print(f'The accuracy of TCA is: {acc:.4f}')"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The accuracy of TCA is: 0.4562\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "68GhdRh4Furf"
      },
      "source": [
        "## Test BDA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5m609OVBFurf",
        "outputId": "8165ebd1-4ebb-49f2-f893-f1e9b24afe10"
      },
      "source": [
        "bda = BDA(kernel_type='primal', dim=30, lamb=1, mu=0.5, mode='BDA', gamma=1, estimate_mu=False)\n",
        "acc, ypre, list_acc = bda.fit_predict(Xs, Ys, Xt, Yt)\n",
        "print(f'The accuracy of BDA is: {acc:.4f}')"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "BDA iteration [1/10]: Acc: 0.4666\n",
            "BDA iteration [2/10]: Acc: 0.4593\n",
            "BDA iteration [3/10]: Acc: 0.4656\n",
            "BDA iteration [4/10]: Acc: 0.4624\n",
            "BDA iteration [5/10]: Acc: 0.4666\n",
            "BDA iteration [6/10]: Acc: 0.4666\n",
            "BDA iteration [7/10]: Acc: 0.4656\n",
            "BDA iteration [8/10]: Acc: 0.4656\n",
            "BDA iteration [9/10]: Acc: 0.4656\n",
            "BDA iteration [10/10]: Acc: 0.4656\n",
            "The accuracy of BDA is: 0.4656\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sp2axlhhILUF"
      },
      "source": [
        "## Test WBDA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "somL-0WnFurf",
        "outputId": "a5ba906f-ee1b-416f-9e7f-b2fbbb4de3de"
      },
      "source": [
        "wbda = BDA(kernel_type='primal', dim=30, lamb=1, mode='WBDA', gamma=1, estimate_mu=False)\n",
        "acc, ypre, list_acc = wbda.fit_predict(Xs, Ys, Xt, Yt)\n",
        "print(f'The accuracy of WBDA is: {acc:.4f}')"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WBDA iteration [1/10]: Acc: 0.4666\n",
            "WBDA iteration [2/10]: Acc: 0.4635\n",
            "WBDA iteration [3/10]: Acc: 0.4520\n",
            "WBDA iteration [4/10]: Acc: 0.4635\n",
            "WBDA iteration [5/10]: Acc: 0.4603\n",
            "WBDA iteration [6/10]: Acc: 0.4624\n",
            "WBDA iteration [7/10]: Acc: 0.4572\n",
            "WBDA iteration [8/10]: Acc: 0.4624\n",
            "WBDA iteration [9/10]: Acc: 0.4614\n",
            "WBDA iteration [10/10]: Acc: 0.4593\n",
            "The accuracy of WBDA is: 0.4593\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h5B5VO8yKBQC"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}